Solving Infinite Horizon Discounted Markov Decision Process Problems for a Range of Discount Factors

In this paper we will assume the following framework. There is a finite state set I, with i E Z as its generic member, 1 < i < m. For each ie Z, there is a finite action set K(i), with k E K(i) as its generic member. For each i E Z, k E K(i), there is a transition probability, p(i, j, k), that if at a decision epoch the state is i E Z, and if action k E K(i) is taken, then the state will be Jo Z at the next decision epoch, and there is an immediate reward, r(i, k), with 0 < r(i, k) < M < co, there is a discount factor r in the interval [0, p], for some fixed p < 1. We will be interested in values of z within this range, and we will parameterize r by a parameter t E [0, 11, so that r = TV. The actual values of t to be studied will depend upon the questions we may wish to answer and how these might efficiently be answered. A pure Markov decision rule is a function 6: I+ K = lJie, K(i), where if i E Z then 6(i) E K(i). We will be concerned with maximizing the infinite horizon discounted rewards, and we do not need to consider either time dependent decision rules, or those which are a function of the complete history of the process up to a specified decision epoch or decision rules which select an action with a specified probability (see van der Wal [4]). A general policy n is an infinite sequence of decision rules which determine an action at each decision epoch, as a function of the past history, with some probability. In the light of the previous remark we need only consider policies of the form 7c = (~5)~, where 6 E A, the set of pure Markov decision rules, and (~3)~ simply means the application of 6 an infinite number of times. If o,(i) is the maximum infinite horizon expected discounted reward, with 303 0022-247X/89 $3.00